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(54) A computer program product for redeye detection 

(57) A computer program product for detecting eye 
color defects of a subject in an image due to flash illumi- 
nation comprises : a computer readable storage 
medium having a computer program stored thereon for 
performing the steps of detecting skin colored regions in 
a digital image; searching the skin colored regions lor 
groups of pixels with color characteristic of redeye 
defect: and correcting color of the pixels based on a 
location of redeye defect found in step (b). 
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Description 



CROSS REFERENCE TO RELATED APPLICATIONS 

[0001] ThisisaContinuatioin'n^Partof Application Serial Mo. 08/919.560. filed August^ 1997. entitled -A Computer 
Program Product For Redeye Detection' by Jay SchildKraut, et al. 

FIELD OF THE INVENTION 

[00021 The invention relates generally to the field of digital image processing and. more particular to a method for 
detecting redeye in digital images. 

BACKGROUND OF THE INVENTION 

[0003] When flash illumination is used tor the capture of an image sometimes the pupils of people in the image appear 
red. This is caused by lightfromtheflaah unit entering the pupil, multiply reflecting off the retina, and finally exiting back 
through the pupil. Because fight is partially absorbed by capillaries in the retina the pupil appears red »n the image. Tnts 
phenomena is r^rred to as Tedey*.* The probability of redeye being observed increases the closer the flash unit * to 
the optical axis of the lens. Therefore, redeye is commonfy observed in images captured by a small camera with an mte- 

20 [r^^commoniy assigned U,S. Patent 6,432.863 describes a user-interactive method for the detection of objects in 
an image that have the color characteristic of redeye. This method automatically detects carxf date redeye pixels based 
on shape coloration and brightness. 

[0005] Although the presently known method of detecting redeye rs satisfactory, rt is not without drawbacks. The 
2s method of U.a Patent 5,432.863 does not determine whether the candidate pixels are located in a face or are part of a 
human eye- 

[00061 Consequently, a need exists for detecting redeye that overcomes the above-described drawbacks. 
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SO 



SUMMARY OF THE INVENTION 



[0007] The present invention is directed to overcoming one or more of the problems set forth above. Briefly summa- 
rized according to one aspect of the present invention, the invention resides in a computer program predict tor detect- 
ing eye color defects of a subject in an image due to flash illumination, comprising : a computer readable storage 
medium having a computer program stored thereon for performing the steps of: (a) detecting sWn colored regions in a 
33 digital image; (b) searching the stfn colored regions for groups of pixels with color characteristic of redeye defect, and 
(c) correcting color of the pixels based on a location of redeye defect found in step (b). 
[0008] It is an object of the present invention to provide a method for automatically detecting redeye defects, 
[0009J it is an object of the present invention to provide a method for determining whether candidate redeye defects 
are part of the human face. 

40 10010] It is also an object of the present invention to provide a method for determining whether caiKfidate redeye 
defects are part of the human eye. . 
[001 1 1 These and other aspects, objects, features and advantages of the present invention will be more clearly under- 
stood and appreciated from a review of the following detailed description of the preferred embodiments and appended 
claims, and by reference to the accompanying drawings. 

45 

BRIEF DESCRIPTION OF THE DRAWINGS 



[0012] 

so Fig. 1 is a diagram illustrating redeye; 

Fig. 2 is an overview flowchart of the software program of the present invention; 

Fig. 3 is a detailed flowchart of the continuous stan colored region determination portion of Fig. 2; 

Fig. 4 is a binary representation of Fig. 1 illustrating shin-colored regions; 

Fig. 5 is a detailed viewed of the individuaJ continues colored regions of Fig. 4; 
ss Fig. 6 is a diagram of ellipses fitted to the views of Fig, 5; 

Fig. 7 illustrates resized candidate face regions; 

Fig. 3 is a diagram of resized ellipses corresponding to the candidate face regions fitted to Fig. 7; 
Fig. 9 is a detailed flowchart of the candidate redeye determination portion of Fig, 2: 
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Fig, 10 illustrates the candidate redeye defects of Fig- 7; 

Fig. 1 1 is a detailed flowchart of tie eye detection portion of Fig. 2; 

Fig. 12 illustrates an eye template, and zone map; 

Fig. 13 illustrates scoring functions of the present invention; 
5 Fig. 14 illustrates an alternative method of locating feces in an image; 

Fig. 15 illustrates scoring functions; 

Fig. 16 illustrates scoring functions; 

Fig. 1 7 illustrates a method of redeye defect pair detection; 

Fig. 18 illustrates different image orientations; and 
10 Fig. 19 illustrates an overview of the present invention. 

DETAILED DESCRIPTION OF THE INVENTION 

[QOi 3] In throwing description, the present invention wfll be described in the preferred emboo^ent^^ft^re 
program. Those skilled in the art will readily recognize thai the equivalent of such software may also be constructed in 

[0014] Fig. 1 is a grayscale image 1 0 of a color image illustrating two pairs of redeyes 20. 
[0015] Referring to Fig. 2. there is illustrated an overview flowchart of the present invention. A color digital image is 
input to the software program residing on a computer system, such (somputer systems being well known in the art Tne 
<^e ^lues of the digrtaiimage are preferably pmporfo 

ture the image by the original scene S2. The program begins by iderrtifying an separate continuous skin colored regions 

[Wq"^e^ng to Fig. 3, there is illustrated a detail flowchart of step S4 in Fig. 2. First, the red. green, and blue val- 
ues of the color image are converted into LS7 color space S4a using the relations: 



1& 



so 



25 



$0 



7 = ^(ft + B-2G) 

35 

where fl, G. and B y are the red, green, and blue code value of a pixel in the color image, respectively. 
[0017] The next step is to build a three-dimensional histogram. In order to reduce the size of the histogram, first, the 
L $ and 7 code values are o^iartfazed by dividing them by8.0xeqrt(3), 2.0. and 2.0. respectively S4b. These quantized 
code values are referred to as L\ S\ and V. Each combination of L\ S\ and T values is referred to as a Tan" of the 
40 histogram The valued the histogram h[L\S\ 7^S4cisequaltomenurnberofpixasinth^ 

code values of L\ S*. and r. An alternative way of stating this is that the histogram tell us the number of pixels in the 
image that fall into each bin. This number is referred to as the value of the bin. 

[Ooi 81 The histogram is smoothed S4d by replacing the value of each bin by a weighted average of the value of that 
bin and the values of immediate neighboring bins. Next, the peak values in the histogram are found S4e and each bin 
46 in the histogram is assign ed S4f the peak value that ts located closest to it Finally, since each pixel in the color imag e 
has been assigned to a bin of the histogram and each bin has been assigned to a peak, a peak is assigned to each 
pixel in the color image S4g. The single band image in which a pixel s code value is equal to the number of the peak 
that it was assigned to is referred to as the segmented image. 

[0019] Continuous regions in the segmented image that have the same code value are likely to correspond to an 
so object or part of an object in the color image. A unique number (label) is assigned to all such regions in the segmented 
image S4h. The numbers are sequentially assigned starting with 1 for the region with the greatest number of pixels. The 
single band image in which code values correspond to the label of the region that the pixel belongs to is called the 
labeled image. 

[0020] The program then decides which of the continuous regions in the segmented image corresponds to a region 
ss in the color image that has a color that is typical of human skin. The average L S. and T code values of each region is 
calculated and. based on this, each region is assigned a score S4i. A high value of P^n indicates that the region 
is of a caJar that is typical of human skin. Alternatively, a low number indicates that the color of the region is atypical of 
sWn, Regfons for which exceeds a threshold 7^ of 0.10 are referred to as skin-colored repjons S4j. 
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[DQ21] One final step is necessary to associate each face in the color image with a single sWn colored region. The 
process described above will often resutt in a single face being associated with more that one skin colored region 
because due to cornplaxfon, shadows, and etc.. the color of the face is not uniform. Two skin colored regions era 
merged into a single skin colored region if two conditions are satisfied $4k. The first condition requires that the two 

B regions be interconnected. A pixel in region / has a connection to region / if a pixel belonging to region ] is one of the 
eight nearest neighbor pixels. A function Q(K j) is calculated which is proportional to the number of connections 
between pixels of region / and /. The function is nomnafeed so that Oft i) is equal to 1 .0. If Qft j) exceeds the threshold 
MinMeverFmctfon regions/and ;wil! be merged into a single region if the second condition is also satisfied, for exam- 
ple a threshold of 0.005 may be used. The second condition is that the distance between the colore of the regions ; and 

10 /given by 

i 



15 



must be less than MaxMergeColorDistance which is set equal to 4O.0. 
[0022] The process of merging skin colored regions begins with the smallest region which, if the two conditions are 
satisfied, is merged whh a larger region. If region / is merged with larger region / it may then happen that region / gets 
merged with an even larger region /c. When this occurs regions /, j\ and k are merged into a single region. Note that 
so regions t and k may be merged together even though the above two conditions are not satisfied for these two regions. 
They are merged because of their mutual connection to region /. 

[0023] The result of skSn color detection is a map of the sWn colored regions in the color image S41 . Areas that are 
not skin colored are given a code value of zero. The separate continuous $Wn colored regions are numbered consecu- 
tively in order of decreasing region size beginning with the number 1 . Fig. 4 shows a map of the sWn colored regi ons in 

[0024] Referring to Rg. 2, and as illustrated in Fig. 5. a sub-map of each sWn colored region is formed by cutting out 
from the sWn map (Rg. 4) the smallest rectangular section that contains all of that skin region S& For example, stan 
region 30b in Fig. 5 corresponds to skin region 80a in Rg. 4. Fig. 5 shows the map of each separate continuous sfan 
colored regions as an individual sub-map. The column and row of the skin map that correspond to the lop left corner of 
so the sub-map are referred to as Coicotout and Row^. respectively. In the sub-map code values of 255 (white) indi- 
cates that the pixel is located at a position at which skin color is present A code value of 0 (blackj indicates the absence 
of skin color. 

[0025] Referring to Fig. 2, and as illustrated in Rg. 6. in the nert step an ellipse 35 is fitted S8 to the individual skin 
color sub-maps found in step S6 (Fig. 5). A method of fitting an ellipse to a binary image is described in Computer and 
33 RrthfttViRjon. Volume 1. by Robed M. Haralick and Linda G. Shapiro, Addison-Wesley (1992), pp. 639-658. A human 
face is approximately elliptical. Therefore, rf the skin color sub-map is of a human face, then the ellipse should fit the 
skin color map we« and the minor axis of the ellipse should approximately equal the width of the face. A measure of the 
fit of an ellipse to the skin color sub-map is given by 



40 



where N is fte number of skin colored pixels (code value 255) in the map, N wt is the number of skin colored pfrels that 
foil outside the ellipse, N m is the number of skin colored pixels that are inside the ellipse and A is the number of pixels 
in the ellipse. A is also referred to as the area of the ellipse. If an of the skin colored pixels are In the ellipse and the 
number of skin colored pixels equals the area of the ellipse then fit is equal to one and the fit is perfect. When skin 
colored pixels fail outside of the ellipse or the area of the ellipse is greater than the number of skin colored pixels inside 
ft then the value of Fit is diminished, if the value of Fit is less than a predetermined value MinBfipseFH which is set 
equal to 0.70 then we conclude that the sWn colored region is not a face and we do not process it further S10. 
[0025] Another indication of whether the skin color sub-map is of a face is the aspect ratio of the ellipse Aspectfiatio 
which is given by 

AspectRatio = 

u mines 

where is the major axis of the ellipse and D mifi0e is the minor axis in pixels. If AspectRatiO is greater than 
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MaxAspectftetfo which is set equal to 3.0 the sWn colored region corresponds to an object in the image that is too long 
and thin to be a face. The program determines that the skin colored region is not a face and does not process « further 
$10. 

[0027] If thestan sub-map has an acceptable degree of fit to an ellipse and the ellipse has an acceptable aspect ratio, 
s the map potentially indicates the position of a face. Next we calculate a resize factor $ proaca/9 which is given by the fol- 
lowing equation 

e AimEyeDistance xFaceWidthEyeDtstanceRatio 

to 

where AimEyeDistance which is set equal to 75 pixels is the desired distance between eyes, and 
Fac&WidthEyeDistanceRatio which is set equal to 2.0 is the ratio between the width and eye distance for a typical face. 
H s r9St(M Is less than MinPrescale 0.10 or greater than MaxPrescale 1.50 the skin colored region is not processed 

is further $1 0- The next step is to cut-out from the color image a sub-color-image that corresponds exactly to the location 
of the sub-map S12. tf the minor axis of the ellipse is approximately equal to the width of the face then the distance 
between the eyes in the face should be close to AimEyeOistance. Fig. 7 shows the sub-color-images 40 after they have 
been resized in this manner, it is instructive to note thai Fig. 7 is illustrated as a gray scale drawing, although the actual 
image is a cofor image. Fig. 8 shows ihe ellipses 50 that correspond to each of these sub-color-images that have also 

to been resized S14. In practice, it is desirable to add extra rows and columns to the edges of the resized sub-color- 
images and sub-maps so that when these images are processed further an out-of-bounds pixel is not addressed The 
top and bottom of the images are padded with Pad rows and the left and right side with Pad columns. 
[00281 Now that skin colored regions that have the shape of a face have been identified, the location of candidate 
redeyes need to be identified S16. which is illustrated in detail in Fig. 9. Now referring to Fig. 9, the sub-color-images 

25 40 are processed so as to identify small red features. The program begins by defining a new single band image S16a 
with pixel values X given fy 

X-R- Max(G r 6) 



30 where R. G, and G. are the red, green, and blue code value of the sub-color-image, respectively. 

[0029] ' Redeyes in the new image will appear as small elliptical areas of high code value possibly with a small low 
code value region in the middle that is due to glint in ft e pupil, "Hie affect of glim is removed by performing a gray scale 
morphotogical dosing Sl6b using a Wjctose x W_dose kernel, for example a 3 x 3 kemal although other sees may 
also be used. Gray scale rixxphologfcaTopei»tions are disclosed in ImflCte Analysis and Mathematical Mprprglra\Jfok 

ss umaX by Jean Serra. Academic Press (1982). pp. 424*78. Next, the small regions of high code value are removed by 
a c/ay scale morphological opening operation using a W_open x Wjopen kernel, for example a 5 x 5 temal although 
other sizes may also be used 816c. The opened Image is then subtracted from the closed image in order to form a 
residual image Sled. This Image shows what was in the opened image, but not in the closed image. Namely, small 
regions of high code value which correspond to small red features in the sub-color-imaga Next the residual image is 

40 smoothed Sl6e wHh a linear filter having the kernel shown below. 



121 
242 
121 

46 

For each pixel in the smoothed residual Image, a 7x7 window centered at that pixel is examined If the code value of 
that pixel exceeds the threshold Tpeak which is set equal to 5 and is greater than or equal to the cede value of all the 
other pixels in the window, that pixel is classified as a peak S1 et. Fig. 1 0 shows the peaks 37 for all of the sub-oolor- 
60 images in Fig. 7. After all the peaks in the smoothed residual image have been found the individual peaks are examined 
Sl6g. First if a pixel has been classified as a peak and a neighboring pixel that is west, north-west, north, or north-east 
of this p«ei has also been classified as a peak the peak is eliminated Sl6h. 

10030] A pixel ftat has been classified as a peak is a cancSdate redeye pixel, it is possible however that the location 
of the peak coincides with glint in the pupil and not the red defect For this reason, pixels within a distance GiintRadfus 
6s equal to 2 from the peak are examined S1 6i. The candidate redeye pixel is moved to the nearby pixel with the highest 
color score P OTtor which wifl be defined below. 

[0031 1 Next, the candidate redeye pixBl is used as a seed to grow a continuous region of pixels of similar color, tf the 
number of pixels in the region is less than MinStee or greater than MaxSize the region is not of a size that is character- 
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tstfc of a redey^ defect and the candidate redeye pixel is eliminated S16j. 

[00321 The result of the above processing is a map of candidate redeye pixels for each sub-color-image Si 6k. The 
ellipses in Rg. s are approximate maps of the region in the corresponding sub-color-images in Fig. 7 that have been 
identified as potentially being a face. Therefore, only the candidate redeye pixels thai fan inside of the ellipse are con- 
s sidered in the next phase eye detection which is outlined in Fig. 1 1- 

[0033] Referring back to Rg. 2. the purpose of eye detection is to determine whether the candidate redeye pixels are 
indeed part of an eye. The eye detection procedure requires a monotone veision of the color image S18. The green 
band of the color image is used after the contrast is increased by transforming the green pixel code values using the 
equation 



G' = 255 




is where G is the code value of the green band and y is a parameter which is set equal to 2.0. This monocolor version of 
the color image will be referred Id as the luminance image; 

[00341 The eye detection procedure S20 in Fig. 2 is based on the process of template matching. It facilitates under- 
standing to note that any image of an eye can be used a the template. The top image 60 in Rg. 12 shows a left-eye 
template. The bottom image 70 shows a division of the template into zones. Zone 1 is the eyebrow region. Zones 2 and 
20 a are the left and right sides of the eye, respectively. Zone 4 includes the pupil and iris. Zone 0 is not used. The eye 
template was taken from an image in which the distance between the eyes is TemplateEyeDistance equal to 306 pixels 
and the tilt of the two eyes is dose to zero. As discussed above, a pair of redeyes in the resized color sub-images should 
be approximately a distance AimEyeDtstence (75 pixels) apart Therefore, in order for the template to be of the proper 
size to match an eye is must be resized byafactorof 

25 

o AlmEyeDistance 
°" Te mpfatsEyeDistsnco 



30 [0035] In practice, the estimation of the face width from the minor axis of the ellipse will not always be accurate. Also, 
the eyes may be tilted. For this reason starting with the original left-eye template and the zone map, a collection of left- 
eye, right-eye (mirror image Of left-eye), and zone maps are generated that span a range of sizes and orientations S22. 
The original eye template and zone map are resized from a factor of S Q x Narrow to S p x Wide in increments of SStep. 
Preferred values of Narrow. Wfde, and Sstep are 1 5, 0.50. and 0.05. respectively In order to accommodate tit for each 

as resize factor, a series of tilted templates and zone maps are generated that range from -MaxTilt degrees (clockwise tift) 
to MaxTilt degrees in increments of TStep degrees S22. The preferred value of MaxTilt is 30 degrees and of TStep is 
2.0 degrees. 

[0036] Referring to Fig. 1 1 . a detailed flowchart of step S20 of Fig. 2 is shown. A pair of candidate redeye pixels are 
considered that hypothetically belong to a left and right redeye pair S20a. The scale of flie eye relative to the original 
to eye template is related to the distance S20b between the candidate redeye pixel pair by the equation 

((i p -rV g »(l r ff / ) 2 ) 1y2 

*petr - TempiateEyeD&ance 

45 

where L p (R p ) is the column of the left (right) candidate redeye pixel, Lj {R f ) is the row of the left (right) candidate redeye 
pixel. (The column numbers begin with 1 and increase from left to right The row numbers begin with 1 and increase 
from top to bottom.) The tilt S20b between the candidate redeye pixels te given by 

[0037] As discussed above, an ensemble of eye templates and zone map templates were made that span a range of 
65 resize factors from S 0 x Narrow to S 0 x Wide with resolution SStep and with a tilt from -MaxTilt degrees to MaxTilt 
degrees with a resolution TStep. The left-eye template, right-eye template, and zone map that most closely match the 
value of Sp^y and Tilt for the pair of candidate redeye pixels is used in the correlation step that follows. If S^ f or TUt 
are outside of this range, this pair is not processed further S20& 
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[003S] After an eye template has been selected the next step is to determine if the region around the redeye pixel 
matches an eye. This fedone by performing a correlation of the left-eye template with a region around the left candidate 
redeye pixel and the right-eye template with a regSon around the right candidate redeye pixel of the luminance image 
$20d. One step of the correlation process is to match up pixete of the template and luminance image and calculate the 

g product of their code values. The center of the template images corresponds to the center of the eya Since the candi- 
date redeye pixels are cfose, but not necessarily at the center of an eye. we perform the correlation several times with 
the center of the template matched to all of the pixels within a square that extends a distance LookAround equal to 3 
about the candidate redeye pixel. The correlation is performed separately for zones 1 through 4 of the template (see 
Rg. 12). These correlations are referred to as Cz1 t Cz2. Cz3, and Cz4. In addition, an overall correlation is calculated 

jo for a region that consists of the sum of 2ones 1 through This overall correlation is referred to as C. The pixel in the 
square around the candidate redeye pixel with the highest value of the overall correlation C is the best guess of the 
center of an eye which contains the candidate redeye pixel. This pixel is referred to as the eye-center pixel. Both the left 
and right candidate redeye pixels have an associated eye-center pixel. 

(003SJ The correlation process is new explained in detail. The template image is denoted by the function * (p r f) where 
is p is the column number and / is the row number. The number of columns and rows in the template is w and h, respec- 
tively. The center of the eye template is approximately the location of the center of the eye. A zone of the template is 
correlated with the luminance image which we denote by T(p. I) at column p Q and row by calculating the product n 

given by. 

z p%z fez 

2s wherep€ Z means that column pis in zone Z, /e Z means that row /is in zone Z, and /v>is the number of pixels in 
the zone. The mean code value of the template in zone Z given by 

36 *p€*/€J 



is also calculated. In addition, the standard deviation of the template in zone Z is calculated according to the equation. 



40 Similarly; we calculate the mean code value of the luminance image in zone Z using the equation 



M r =^ £ £r(p+p 0 -*/2-W o -/?V2-1) 

z pez /ez 

45 

and the standard deviation using the following equation 

^ z pez/zz 



[0040] Using the quantities defined above the correlation of the luminance image with the template in zone Z is given 
55 by the relation 



2 1/2 
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s If the code values of the image and the template are exactly the same in zone Zthen O z is equal to 1.0. If the image 
and the template are completely uncorrelated then Q z wiD be equal to zera 

[0041] The values of C. 0*. and C z4 for the eye-center pixels are used in the calculation of a score that is 

a measure of the liKeiihood that the pair of candidate redeye pixels are part of a redeye defect in the sutxtttonmage 
saoe. Each of the correlations are used as a variable in an associated scoring function that ranges from 0.0 to 1 .0. For 
10 ©cample, the scoring function associated with the overall correlation C which we refer to as pC(C) is 0.0 if the value of 
C tor an eye-center pixel indicates that it is very unlikely that the pixel actually is tocated at the center of an eye. On the 
other hand if the value of 0 is in a range that is typical of the correlation of the template with an eye then pC(C) is m 
Otherwise pCfCJ takes on an intermediate value. The scoring function pC(C) and other scoring functions described 
below are shown in Fig. 13. 

is [00421 Scores are defined based on these scoring functions which will be combined later into an overall score for a 
candidate redeye pair. The following equation defines a score related to the overall correlation C as simply 

so The score Pzone associated with the zone correlations is a weighted average of the zone correlation scoring functions. 
It has been found that the correlation in zone 4 (the pupil) is a much more reliable indicator of the presence of an eye 
than the other zones. For this reason it is given more weight than other zones. Typical we set the weight W equal to 6.0. 
Pjwis given by 

25 pC 2 ,(C 2 ^pO z2 (C x2 )*pG^(C z ^WpG r 4C ze ) 



10043] it has been found that the standard deviation of the luminance image thai was calculated in the process of 
so calculating the overall correlation C is a good indicator if the feature in the luminance image centered at the eyecenter 
pixel is actually an eye. For instance, if is very low than the feature is of too low contrast to be an eye. With this in 
mind we define a score associated with cy by 

95 

[0044] Finally, the color of the candidate redeye pixel must be indicative of a real redeye defect For this calculation 
the red, green, and blue, code values of the candidate redeye pixel is convert into luminance (Lvm), hue (Hue), and 
saturation (Sat) values. Luminance is calculated as follows 

40 U]] _ Max(R.G t B) + Min(KG.B) 



The value of Lum for a pixel ranges from zero to the highest possible code value. The saturation given by 

so is a value ranging from 0 to 1 Q0_ The hue is defined as in Computer Gra nnies Prindples and Practice 2nd ed.. Addison- 
Wesley Publishing Company, page 593, except the color red is shifted to a hue angle of 120 degrees. The value of Hue 
may rangefrom 0 to 360 degrees. The score that is related to the color of the candidate redeye pixel is defined by 

Pcotor = pL(Lum)pH(Hue)pS($8t) 

SB 

[0045] The result is a score which indicates the likelihood that a candidate redeye pixel is actually part of a red- 
eye defect in the image. This score is defined by 
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lis value is in the range of 0.0 to 1 .0. The figure of merit is calculated for both the left and the right candidate redeye 
pixels in a pair. The average of these two values is given by 



w 



SO 



The pair of candidate redeye pixels for which P^ is the largest is referred to as the best pair of candidate redeye pixels 
S20f. If Ppsir exceeds the threshold MinEyeScots equal to 0.05, then the program processes further. Otherwise, the 
program concludes that a pair of redeyes is not present in the sub-color-image S20g. 

[0046] ft is important to minimize the possibility that the best pair of candidate redeye pixels that are nor part of a pair 
ib of eyes with a redeye defect in the color image be incorrectly classified. One method of confirming that a pair of redeyes 
has indeed been located is to use the fact that a human face is approximately symmetric about a line that bisects the 
face S24 in Fig. 2. In order to do this, the sub-color-image is rotated so that the tilt of a line connecting the best pair of 
candidate redeye pixels is equal to zero. Next, an image centered at the midpoint between the eyes is cut-out of the 
sutxolor-image. This image has a width of i JS times the distance between the candidate redeye pixels and a height 
go equal to a quarter of its width. This image is in turn cut in half. The left half-image we refer to as £=W/>/'J and the right 
half-image by FrighiM where the superscript x refers to a band of the color image For example, EWk/j refers to the 
red band of the image. The columns in the right hatf-image are inverted (the first column becomes the last column, etc) 
so that it becomes a mirror image of teelf. A correlation of and is performed by first calculating the 

sum of products 

2S 

p t 



where the summations over p and / are over all of the columns and rows in the half- images, respectively, and N is the 
number of pixels in the half-images. The correlation is given by 



~x , ft sym' M tott M mt 
° toft* rfght 

where A^fefr and bright are the mean code values of band x of the half-images and e*left and bright are the stand- 
40 ard deviations. A score is defined based on a symmetry scoring function pSymf&sym) by 

Psfln = pSym(C f sym)p$ym(C 9 sy™)pSym(C* sym) 

as [0047] The final score P is simply the product of P^ and P^ 

p ■ p sym p peir 

so if this score which may range between 0.0 and 1-0 exceeds a threshold MinScofe, which is set equal to 0.05 $26, then 
the candidate redeye pixel pair is assumed to mark the location of a pair of redeye defects in the resized sub-color- 
image. 

[00481 finally, the positions of the left and right redeye defects in the original color image are calculated based on the 
position of the left and right candidate redeye pixels in the resized sub-color-image using the relations 

55 

-Pad 
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s where p and / are the column and row of the left candidate redeye pixel in the resized sub-cotor-image and p and r are 
the corresponding positions in the original color image S28. 

[0049] It sometimes happens that two different skin colored regions after being fitted to an ellipse will overlap or be 
very close together This may result in the same redeye pair being found twice or the detection of two redeye pairs that 
are too close together for both to be truly a pair of redeyes- For this reason, after all the redeye pairs in the color image 
10 have been located it is determined if any two pairs have redeye locations less than MintnteipairEyeDistance equal to 
20 pixels apart rf this is the case the pair with the lower score is eliminated S30. 

[0050] On page 5 line 22 to page 8 tine 22, a procedure is described that determines the location of possibTe faces in 
the image (see Figure 1 . In this procedure the color and shape of regions Of approximately uniform color are examined. 
There is an additional means of verifying that skin colored region is a face. This method is based on the observation 

is that a face is usually sWn except for the eyes, eyelashes, and mouth. This is illustrated by figure 5 which shows maps 
of continuous sWn colored reports. Notice that there are significant holes within the skin map in which are located pixels 
that do not have the color of skin. These holes occur where the eyes, eyelashes, and mouth are located. There are 
many situations where the eyelashes and mouth will not cause a hole in the skin map and situations where other image 
features will cause additional holes. Even though the number of non-skin colored holes may vary for marry reasons rtjs 

so useful to place a limit or assign a scoring function based on the number of holes. Typically, if the number of non-skin 
colored holes is greater than 1 0 the continuous skin colored region is determined not to be a fee© and is not processed 

WW] In figure 1 4, an alternative of Fig. 9 steps S26f through Si 6k is illustrated for finding si ngle redeye defects. As 
a result of steps Si 6a to S16e in figure 9 a smooth residual image is produced in which pixels that belong to small red 

25 features in the original image are assigned a high code value and all other features are assigned a low code value. 
Starting in step S24a infigure 14 the algorithm examines the peaks in tha smooth residual image. 
roQ52l When a map of the sMn colored regions of a fece is made as in figure 5 the eyes win net bs part of the map 
because the colors normally found in an eye are different than those normally found in skin. Therefore, in step S24t> it 
is determined rf the peak pixel is part of the skin colored region of the image. If this pixel is part of the skin colored 

so region, the program proceeds to the next peak. H the peak pixel is not in the skin colored region, the program proceeds 
to the next step S24c 

[00531 Aredeyedefectmustoocurintheregionofaface, Usually it will occur within a hole in the skin mapoftheface. 
However. rF the fece is in a slight profile position with respect to the camera that captured the image, the redeye defect 
may occur outside of the skin map. As shown in step S8 of figure 2. the map of skin colored regions is fitted to an ellipse. 
36 This ellipse is referred to as the ellipse of best fit It is possible for a redeye defect that is associated with a face to fall 
outside the ellipse of best fit due to the face being in a prof ne position. For this reason, an expanded ellipse is defined 
that has the same center and shape as the ellipse of best fit but a greater radius. Typically the radius win be twice as 
large. A peak pixel must fall inside of the extended ellipse for ft to be considered further. This is shown in step S24c of 
figure 14- 

40 10054] A peak pixel may be located at a position in a redeye defect that corresponds to the glint caused by the camera 
flash reflected form the eye. In this case, the color of the pixel will be a bright neutral. It is desirable to locate a pixel near 
the peak pixel that has the color of the redeye defect For this reason in step S24d pixels in a radius of GXntRadhis 
around the peak pixel are examined, rf no pixels are found with a non-zero probability {described herafnbelow) based 
on color of being part of a redeye defect, move on to the next peak pixel. Otherwise, select 

46 [0055] This pixel selected in step S24d is referred to as the grow pixd because it will be used to grow a region of the 
image that corresponds to the complete redeye defect But first ni step S24e, it is determined if the grow pixel has 
already been processed. If so, consider the next peak pixel. 

[0056] In step S24f, the grow pixel is used as a seed to grow a connected region of pixels aD of which have a color 
that is within some set odor distance of the grow pixel and has a non-zero redeye color probability. This grown region 

so is a carxfidate redeye defect Referring to step S24g. the average color in the candidate redeye defect is calculated. 
[0057] Trie next series of steps involve assigning a score to the candidate redeye defect based on several character- 
istics. The first is the code value of the peak pixel in the residual image that was used to locate the candidate redeye 
defect This code valuets normalized by dividing it by the maximum code value possible in the originaJ image. For exam- 
ple, divide by 255 in the case of an 8-bit image. The normalised code value is a assigned a score S24h according to the 

es scoring function shown in figure 15a. As is obvious from the graph, if the normalized peak height is between 0 to 0.02 
the score is zero; for a normalized peak height above 0.02 to 0.1 the score increase linearly; and for a normalized peak 
height between 0.1 to 1 .0, the score is 1 . t 
[00581 Similarly, a score is assigned based on the size (number of pixels) in the cancfidate redeye defect S24i. The 
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scoring function associated with this quantity is also shown in figure 15b> 

[00591 Referring back to figure 14, in step S24j a score is assigned to the defect based on its average color. Approx- 
imately 200 images, were examined which had one or more redeye detects in them. By sampling redeye pixels a joint 
prcoab% function was buitt denoted by p(n?/K H. Sat) which is proportional to the probability of a redeye defectpixel 
s having a luminance V, hue H, and saturation Sat, This function is normalized so that its maximum value equals 1 .0. 
The candidate redeye defect is assigned a score based on the value of pfre / Y 9 H. S&) for the average color of the 
defect. A typical scoring function for / Y t H m Sat) is shown in figure 1 Sa. 

[0060] Referring back to figure 14, in the next step SZ4k, the candidate redeye defect is fitted to an ellipse. In step 
S241 , a score is assigned to the candidate redeye defect based on the aspect ratio of the ellipse. A typical scoring func- 
10 tion for the aspect ratio Is shown In figure 1 6b. In step S24m a score is assigned for the quality of the fit to an ellipse. A 
typical scoring function for the quality of fit to elGpse is shown in figure 16c. 

T0061] A total score can now be assigned S24n to me candidate redeye defect based on a combination of an of the 
above scores. Typically, this score is the product of the individual scores although the use of the sum or median value 
Is ateo contemplated The candidate redeye detects can now be assigned a ranked order based on this score. 

is [0062] inwbseo^ steps that inw^^ 

didate redeyedefects that have the highest score first and to place a limit on the total number of candidate redeyedefect 
pairs that can be examined by the algorithm, "mis prevents the execution time of the algorithm from becoming undesir- 
ably large in the case that there is a large number of candidate redeye defects and insures that the best candidates will 
be examined before the limit is exceeded. 

20 [0063] Although a major intent of this invention is to find redeye pairs sometimes redeye defects occur singly because 
one eye rs obscured or lor some reason does not exhibit the redeye phenomena The procedure at this point is suited 
for finding single redeye defects because at this point no use has been made of the fact that redeye defects normally 

occur in pairs. „ 
[0064] in figure 1 7 a method is shown of processing two candidate redeye defects at a time in order to determine if 

sb they are in fact a pair of redeye defects. This method is an alternative ernbodirnent to the method shown in figure 1 1 . It 
should be understood that there are many features in an image that may be mistaken for a redeye defect For example, 
red ornaments on a Christmas free. The key to being able to detect redeye defects without human imerventlog which 
is a major objective of this invention, is to require that redeye defects occur in pairs and that the two defects in the pair 
be consistent with one another and with the skin colored region near or in they are located. 

so [00651 In step S30a in figure 1 7 the program starts with the candidate redeye defects that were obtained by the series 
of steps shown infigure 14 At this point, each candidate has been assigned a score based on its color, size, and shape 
which relates to the probability that the candidate is a real redeye defect to step S30b the program starts by selecting 
a candidate redeye delect as the left candidate redeye defect and pairs it up with another candidate defect referred to 
as the right candidate redeye defect At this point, the program hypothesizes that the left candidate is the redeye defect 

36 on the left when viewing an upright face in the image and the right candidate is the redeye defect on the right 

[0066] In step S30c the program determines if the distance between the left and right defect m the original image in 
units of pixels is at least MfnEyeDtstanoe which is typically set to 30. If the distance is less than this value even if the 
candidates are actual redeye defects they will be so small that they will not be very visible. Therefore, we stop process- 
ing this pair and go to the next pair. 

40 [0067] At this point as described on page 8 line 23 to page 9 line 16, the image has been resized so that if the skm 
colored region is indeed a face the distance between eyes will be approximately equal to AimEyeDtstanc* which is typ- 
ically 75 pixels. In step S30d the pair is assigned a score based on the ratio of thefr separation to AimEyaDistimce. 
[006S] A powerful means of determining if a pair of candidate redeye defects are actually redeyes is to determine If 
their properties are consistent This will be illustrated in detail below. For this purpose a measure of the consistency C 

45 of a quality X is defined by 



50 



Eq.34 



where X f is the value of the quantity for the left candidate redeye defect and X, is the value forthe right candidate defect. 
The smaller the value of C the more consistent the quantity X is. 

[0069] The odor of a redeye defect may vary considerably from image to image mainly because of color balance dif- 
ferences. Also, redeye defect color variations occur among people. However, when two redeye defects occur in a face 
55 their color should be consistent with each other. In step S30e in figure 1 7 the program assigns a score to the candidate 
redeye defect pair based on the consistency of their color. In general, the score decreases as the value of C increases. 
[0070] The size of the left and right candidates defects should be roughly the same In step S30f a score is assigned 
based on the consistency of the sf2e of the left and right defect 
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[0071 J Given the distance between the eyes in a face it Is possible to place limits on the size of the pupils of the eyes. 
Using the distance between the left and right candidate redeye delects a nominal pupil size is calculated For example, 
rf the number of pixels between the eyes ismultplied by 0.083, the square of the product is roughly the number of pixels 
in a pupil. Of course pupil size may vary significantly as a function of ambient fight levels and other factors, but the 
fi number of pixels in a redeye defect should approximately match the number of pixels in a pupil of nominal size within a 
factor of approximately three. In step S30g, a score is assigned to both the left and right defect based on the ratio of 
their sizB to the nominal pupil size. 

[0O72J In most cases the orientation of the image is unknown. For example, the picture may be taken with the camera 
held upright and the image will have a landscape orientation as shown in figure 18a, Alternatively, the camera may be 

to held sideways and the image will have a portrait orientation as shown in figure 18b and figure 18c In addition to camera 
orientation the pose of a person in the image affects the orientation of a pair of redeye defects. For example, the Image 
may be captured with the camera upright, but a person lying down may have their eyes oriented as in figure 1 8b. 
[0073] The purpose of step S30h in figure 1 7 is to infer the orientation of the redeye defect pair that may be associated 
with a pair of candidate redeye defects. Referring to figure 1 8a the angle 0 is defined as the angle between a horizontal 

ts line a that passes through the left candidate redeye defect and a line b that passes through both candidate redeye 
defects. 8 is positive if line b is orientated counterdoclwise from line a. If the absolute value of 6 is less than MaxTih 
which is typically set to 30 degrees the orientation is assumed to be as In image figure 18a. If the absolute value of e + 
90 is less than MaxTVt the orientation is assumed to be as shown in image as in figure 18b. Finally, if the absolute value 
of 8 - 9Q is less than M&Ttlt the orientation fe assumed to be as in image as in figure 18a it should be noted that the 

20 object of this invention is to detect a pair of redeye defects and not to determine which defect is in the left and right eye. 
Therefore, it is of no consequence if a defect pair wfth an orientation as shown in image in figure 18b is mistaken tor a 
pair with the orientation as shown in figure 18c 

[0074J If 9 satisfies one of the three conditions stated above, the tilt of the candidate redeye pair with respect to a 
horizontal line in the image is less that MaxTat. In this case {see step S30i in figure 17) the pair of candidate redeye 

25 pixels is processed farther. Otherwise, the next pair is considered. 

[0075] In step S30j in figure 1 7 the region around the left and right candidate redeye defect are correlated with a left 
and right eye template as deserved in detail hereinabove on psgs 11, line 18 through page 13, line 3. The iempiates 
are chosen based on the assumed orientation, distance between the candidate redeye defects, and their relative tilt A 
score is assigned based on the value of the correlations. 

so [0076] In step S30k a score is assigned base on the consistency of the standard deviation of the luminance in the 
repjon around the left and right candidate redeye defects, As described hereinabove on page 23, line 6 through page 
1 5 line 21 , this standard deviation is calculated in the process of calculating the correlation. 
10077] Finally, in step S301 the total score for the candidate redeye defect pair is calculated The total score is pref- 
erably some combination of all of the scores mentioned above. For example, the total score is equal to the product of 

35 the individual scores. After this step, the program returns to step S30b and repeats the process for the next pair of can- 
didate redeye defects, in some cases, it is desirable to place a limit on the total number of pairs that can be processed 
so that the execution time does not become too large in the case that there is a very large number of candidate redeye 
defects. 

[0078] The candidate redeye defect pair with the highest score is processed further. The mirror symmetry of the 
40 region of the image in the vicinity of the pair about a line that bisects a line that connects the candidate redeye defects 
is calculated. This is discussed in detail starting on line 18 on page 21 . A score is assigned to the pair based to sym- 
metry. 

[0079] An additional method of verifying that the pair is an actual pair of redeye defects is based on the fact that peo- 
ple have no more than two eyes. This means that a pair of candidate redeye defects should not have additional image 
4S features nearby that are similar to the defects in the pair. A score is assigned to the pair of candidate redeye defecte 
based on the number of other candidate redeye defects that are nearby. The score decreases as the number of nearly 
additional candidate increases. 

[0080] The modified score is then calculated which is a combination of the score calculated in step S301 and the sym- 
metry score and extra candidate defect scores described above. Preferably, the scores are combined by multiplication. 
so If the resultant score exceeds some threshold the pair of candidate redeye pixels are classified as a pair of redeye 
defects and have their color corrected. 

[0081 j with reference to fig. 19, there is shown an overview of the presemirrvento 

processed 200 to detect the redeye defects. The results 300 are used to selectively modify the image 400 at the loca- 
tions obtained in the detection step. The result is a modified digital image 500 with some or all of the redeye defects 
ss corrected- 
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Claims 

1 . A method for detecting eye color detects of a subject in an image due to flash Biuminatron. the method comprising 
the steps of: 

5 

(a) detecting a skin colored region In a digital Image; 

{b} determinin0 if the skin colored region has a predetermined characteristic of a human face; 
(c) detecting a pair of candidate redeye defects in or adjacent to the human face based on comparing a char- 
acteristic of the candidate redeye defects to a characteristic Of the detected human face; and 
io (d) selecting the candidate redeye defects as actual redeye defects based on the results of step (c). 

2. The method as in claim 1, wherein step (c) includes detecting the pair of redeye defects based on a comparison of 
a distance between the candidate defects and a dimension of the human face- 
re 3. The method as in daim 2, wherein step (c) includes piwiti^ 
human face, 

4. A method for detecting eye cotor defects of a subject in an image due to flash illumination, the method comprising 
the steps of 

20 

(a) detecting a pair of redeye defects based on a characteristic of each defect in the pair in relation to a dis- 
tance between the redeye defects. 

5. A method far detecting and correcting eye color defects ot a subject in an image due to flash illumination, the 
£5 method comprising the steps of: 

(a) detecting a skin colored region in a digital image: 

(b) determining if the skin colored region has a predetermined characteristic of a human face; 

(c) detecting a pair of candidate redeye defects in or acQacent to the human face based on comparing a char- 
30 acteristfc of the candidate redeye detects to a characteristic of the detected human face; 

(d) selecting the candidate redeye defects as actual redeye defects based on the results of step (c); and 

(e) correcting the redeye defects. 

6. The method as in claim 5, wherein step (c) includes detecting tte pair of redeye defects based on a comparison of 
& a distance between the candidate defects and a dimension of the human face. 

7. The method as in claim 6, wherein step (c) includes providing a width of the human face as the dimension of the 
human faca 

40 8. A method for detecting and correcting eye color defects of a subject in an image due to flash illumination, the 
method comprising the steps ot 

(a) detecting a pair of redeye defects based on a characteristic of each defect in the pair in relation to orienta- 
tion of the defect pain and 
45 (b) correcting the pair of redeye defects. 

S. The method as in claim 8, wherein step (a) includes providing a match to an eye template as the characteristic. 

1 0. The method as in claim a further comprising determining whether the pair of defects are in a landscape or porfrait 
so orientation based on the orientation. 
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